Some issues such as high computational complexity and low recognition accuracy still exist in human interaction recognition. In order to solve those problems, an innovative and effective method based on statistical features of key frame feature library was proposed. Firstly, features of global GIST and regional Histogram of Oriented Gradient (HOG) were extracted from the pre-processed videos. Secondly, training videos with different kind of actions were clustered by the k-means algorithm respectively to get key frame feature of each action for constructing key frame feature library; in addition, similarity measure was utilized to calculate the frequency of different key frames in every interactive video, then the statistical histogram representation of interactive videos were obtained. Finally, the decision level fusion was achieved by using Support Vector Machine (SVM) classifier based on histogram intersection kernel to obtain impressive results on UT-interaction dataset. The experimental results on standard database show that the correct recognition rate of the proposed method is 85%, which indicates that the proposed method is simple and effective.